Skip to content

ci: extend go1.26.4 pin to remaining test workflows (follow-up to #5056)#5066

Merged
darccio merged 1 commit into
mainfrom
ci/extend-go-1.26.4-pin
Jul 24, 2026
Merged

ci: extend go1.26.4 pin to remaining test workflows (follow-up to #5056)#5066
darccio merged 1 commit into
mainfrom
ci/extend-go-1.26.4-pin

Conversation

@kakkoyun

@kakkoyun kakkoyun commented Jul 24, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Follow-up to #5056, which pinned Go 1.26.4 (off corruption-prone 1.26.5, golang/go#77168) but only in smoke-tests.yml + govulncheck-fix.yml. The remaining test/build workflows still ran 1.26.5 via stable/"1.26", so the same zip: checksum error kept hitting the default-branch and PR test matrix.

Extends "1.26.4" to the test/build workflows still exposed:

  • main-branch-tests.yml — the default-branch test driver
  • pull-request.yml
  • appsec.yml — macOS + container matrices and two stable jobs (the container matrix also feeds the golang:1.26.x-* image tags)
  • dynamic-checks.yml
  • orchestrion.yml

oldstable (1.25.x) and "1.25" entries are unchanged.

govulncheck.yml is deliberately left on stable. govulncheck scans the toolchain's stdlib, and 1.26.4 still carries GO-2026-5856 (crypto/tls ECH, CVE-2026-42505, fixed in 1.26.5) — pinning it down reintroduces a real finding. Test/build jobs only need to dodge the corruption, so 1.26.4 is fine for them; the vulnerability scanner has to stay on the latest patch.

Data

On smoke-tests.yml (pinned by #5056), the corruption rate dropped from roughly 40% of runs on 1.26.5 to about 10% on 1.26.4 — a reduction, not a fix. 1.26.4 still fails occasionally (e.g. the 2026-07-24 smoke nightly). The residual is expected until Go ships the #77168 fix in a later 1.26.x.

Motivation

Reduce default-branch and PR CI flakiness (incident #57895) by removing the remaining exposure to the go1.26.5 build/module-cache corruption. This is temporary: revert to stable/"1.26" once a fixed Go release is out. CI-only; no code or go.mod changes. Verified with actionlint.

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 24, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 62.84% (+11.43%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7007b07 | Docs | Datadog PR Page | Give us feedback!

@kakkoyun
kakkoyun requested a review from darccio July 24, 2026 11:18
@kakkoyun
kakkoyun marked this pull request as ready for review July 24, 2026 11:18
@kakkoyun
kakkoyun requested review from a team as code owners July 24, 2026 11:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2b2ce03fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version: stable
go-version: "1.26.4"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Pin the Orchestrion matrix output

Pinning this setup step does not pin the Orchestrion integration-test matrix: the next step runs go run ./internal/orchestrion/matrix, whose output is still oldstable/stable (internal/orchestrion/matrix/matrix.go:51), and integration-test uses that output as matrix.go-version (.github/workflows/orchestrion.yml:120). Per the setup-go docs, stable tracks the latest stable Go, currently 1.26.5, so these jobs will still run the unpinned patch rather than 1.26.4; change the matrix output while this pin is needed.

Useful? React with 👍 / 👎.

strategy:
matrix:
go-version: [ "1.25", "1.26" ]
go-version: [ "1.25", "1.26.4" ]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Propagate the pin into unit setup

This pinned PR leg still invokes unit-integration-tests.yml, whose prerequisite set-up job installs go-version: stable before computing the contrib matrix (.github/workflows/unit-integration-tests.yml:47-56). Because setup-go resolves stable to the latest stable patch, currently 1.26.5, PR/main/dynamic workflows can still fail in this setup phase under the patch this change is trying to avoid; pass the requested Go version into that setup step as well.

Useful? React with 👍 / 👎.

- uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version: stable
go-version: "1.26.4"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Refresh the AppSec module cache for the pinned toolchain

When the AppSec module cache has already been populated by a pre-pin run, this job will still reuse it: the cache key is only go-pkg-mod-${{ hashFiles('**/go.sum') }}, and the lookup-only hit skips re-downloading before the test jobs restore that same cache. Since this pin is meant to avoid Go 1.26.5 module-cache corruption, leaving the key unchanged means a corrupted 1.26.5 cache can continue to break AppSec CI even though this step installs 1.26.4; include the Go patch/toolchain in the cache key or force a one-time cache refresh.

Useful? React with 👍 / 👎.

#5056 pinned Go 1.26.4 (off corruption-prone 1.26.5, golang/go#77168) only in smoke-tests.yml and govulncheck-fix.yml. The remaining test/build workflows still ran 1.26.5 via stable or the "1.26" minor, so the same zip: checksum error corruption kept hitting the default-branch and PR test matrix.

Pin the 1.26.5-exposed go-version entries to "1.26.4" in main-branch-tests, pull-request, appsec, dynamic-checks, and orchestrion; oldstable and "1.25" unchanged. govulncheck.yml stays on stable on purpose: govulncheck scans the toolchain stdlib, and 1.26.4 still carries GO-2026-5856 (crypto/tls ECH, CVE-2026-42505, fixed in 1.26.5), so pinning it down reintroduces a real finding. Test/build jobs only need to dodge the corruption, so 1.26.4 is fine there.

On smoke-tests the pin cut the corruption rate from ~40% to ~10% -- a reduction, not a fix. Temporary until Go ships the #77168 fix.
@kakkoyun
kakkoyun force-pushed the ci/extend-go-1.26.4-pin branch from f2b2ce0 to 7007b07 Compare July 24, 2026 11:27
@pr-commenter

pr-commenter Bot commented Jul 24, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-24 11:56:28

Comparing candidate commit 7007b07 in PR branch ci/extend-go-1.26.4-pin with baseline commit cf24b81 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 326 metrics, 0 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

@darccio
darccio merged commit b531af2 into main Jul 24, 2026
337 of 338 checks passed
@darccio
darccio deleted the ci/extend-go-1.26.4-pin branch July 24, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants